111 1 111 urn minimi i 

(id EP 0 816 983 A2 


(12) EUROPEAN PATENT APPLICATION 


(43) 

Date of publication: 

(51) IntCI. 6 : G06F 3/00 


UA.ui.iyyo Bulletin iyyo/u^ 


Application number: 97304473.8 


(22) 

Date OT Tiling. 25.06. 1997 


(84) 

Designated Contracting States: 

• Nielsen, Jakob 


AT BE CH DE DK ES Fl FR GB GR IE IT LI LU MC 

Atherton, California 94027 (US) 


NL PT SE 

• Glass, Robert 



Los Gatos, California 95030 (US) 

(30) 

Priority: 25.06.1996 US 670057 




(74) Representative: Hogg, Jeffery Keith et al 

(71) 

Applicant: SUN MICROSYSTEMS, INC. 

Withers & Rogers 


Mountain View, CA 94043 (US) 

4 Dyer's Buildings 



Holborn 

(72) 

Inventors: 

London EC1N2JT (GB) 

• 

Tognazzini, Bruce 



Woodside, California 94062 (US) 


(54) 

Method and apparatus for eyetrack-driven text enlargement 

(57) 

Apparatus, methods, systems and computer 

display device has the user's interest by using a gaze- 


program products are disclosed that automatically ex- tracking device to monitor where the user looks on the 

pandsthe computer displayed information that interests display. Alter determining which area interests the user 

a computer user. Thus making the information easier for the invention expands the display of the information in 

the user to see. The computer detects which area on a that area. 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 


CM 
< 

CO 
00 
O) 

CD 
CO 

o 

Q_ 
LU 



FIGURE 13 


Printed by Jouve, 75001 PARIS (FR) 


1 


EP 0 816 983 A2 


2 


Description 

Background of the Invention 

Field of the invention 

This invention relates generally to the field of com- 
puter-human user interface technology and more par- 
ticularly to a method, apparatus, system and computer 
program product for allowing a computer to automati- 
cally determine what aspect of the computer's operation 
has the user's interest and to optimize that aspect. 

Background 

Human/Computer Interaction 

An important characteristic of modern computing 
systems is the interface between the human user and 
the computer. Early interactive interfaces were text 
based wherein a user communicated with the computer 
by typing a sequence of characters on a keyboard and 
the computer communicated with the user by displaying 
characters on an output device-common ly a display 
screen. These input characters specified a command to 
the computer's operating system or to an application 
program executing on the computer. This command in- 
voked program logic to perform a given operation. Mod- 
ern computer systems use a graphical user interface 
(GUI) to simplify the interaction between a user and a 
computer. A GUI equipped computer communicates 
with a user by displaying graphics, including text and 
icons ; on a display screen and the user communicates 
with the machine both by typing in textual information in 
response to dialogs and by manipulating the displayed 
icons with a pointing device, such as a mouse. 

Many modern GUIs provide a window environment. 
In a typical window environment the graphical display 
portrayed on the display screen is arranged to resemble 
the surface of an electronic "desktop" and each appli- 
cation program running on the computer is represented 
as one or more electronic "paper sheets" displayed as 
rectangular regions on the display screen. These rec- 
tangular regions are called "windows". Each window 
may include a multitude of panes. Each pane being an 
area for a particular type of information (textual, still im- 
age, moving image, etc.). 

Each window displays information generated by an 
associated application or system program. Further, 
there may be several windows simultaneously present 
on the desktop with each containing information gener- 
ated by a program. A program presents information to 
the user through each window by drawing or "painting" 
images, graphics or text within the window. The user can 
also move a window to a different location on the display 
screen and change its size and appearance to arrange 
the desktop in a convenient manner. The user commu- 
nicates with the program by "pointing at" objects dis- 


played in the window with a cursor controlled by a point- 
ing device and manipulating the objects as desired. In 
some cases the program requests additional informa- 
tion from the user in response to a manipulation. This 
5 request is presented as a "dialog" that allows the user 
to provide the requested information to the dialog from 
the keyboard. 

Each window typically includes a number of stand- 
ard graphical objects such as sizing boxes, buttons and 
scroll bars. These features represent user interface con- 
trols that the user can manipulate with the pointing de- 
vice. When the controls are selected or manipulated, the 
GUI invokes program logic in the underlying program to 
effect a corresponding command. 

One characteristic of a GUI is that the GUI is only 
responsive to a user's explicit manipulation of the point- 
ing device or keyboard. In the case of a mouse, the user 
physically moves the mouse device and a cursor on the 
display moves accordingly. Some pointing devices ac- 
tually track the user's gaze and move the cursor to 
where the user "looks" on the display screen. However, 
even with the gaze tracking (eye tracking) devices, the 
GUI only responds to the user's explicit commands 
whether that command be a button press, a blink, or a 
shift of view. The computer remains a tool that the user 
operates by issuing explicit commands. 

In contrast, humans have the ability to make infer- 
ences by looking at another human's eyes. Pupils dilate 
when people see something attractive. People look at 
what they are interested in and stare at things they find 
interesting. Also ; human eye movements reflect thought 
processes. Thus, humans observe what other persons 
do with their eyes and make inferences as to what that 
other person is interested in and/or thinking. 

The prior art in computer-human interfaces does 
not determine the user's immediate interest. Prior art 
computer-human interfaces simply respond to a user's 
command, whether input by typing the command at a 
keyboard, by manipulating a mouse to move a cursor, 
or by using a gaze tracking device to move a cursor. 
Thus, the computer is unable to detect or anticipate what 
characteristic of the computer's operation is of most in- 
terest to the user at any given time. 

Gaze Tracking Devices 

Most gaze tracking devices operate based upon the 
principal that the direction of a person's gaze is directly 
related to the relative positions of the pupil and the re- 
flection of an object off the cornea (gaze tracking is often 
termed eye tracking). These devices often include im- 
age processing capabilities that operate on a video im- 
age of an eye to determine the gaze direction of the eye. 
These image processing capabilities are enhanced by 
using the bright eye affect. 

The bright eye affect is a result of the highly reflec- 
tive nature of the retina This characteristic of the retina 
means that a significant amount of the light that enters 
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an eye is reflected back through the pupil. Thus, when 
light shines into an eye along the axis of a camera lens, 
the retina reflects a significant portion of the light back 
to the camera. Hence, the pupil appears as a bright disk 
to the camera. This affect allows the pupil to be more 
readily imaged from a video of an eye. 

Other methods exist for gaze tracking. Some incor- 
porate having two video cameras, one for tracking head 
movement and the other for measuring a reflection off 
of the eyes. Other mechanisms involve measuring elec- 
tric potential differences between locations on different 
sides of an eye. High accuracy devices are very intru- 
sive on the user and require that the user's head be held 
in a fixed position or that the user wear special equip- 
ment to track the eye. 

Recently, an eyegaze eyetracking system has been 
developed as described in The Eyegaze Eyetracking 
System - Unique Example of a Multiple-Use Technology 
4th Annual 1994 IEEE Dual-Use Technologies and Ap- 
plications Conference, May, 1994. This system compris- 
es a video camera located below a computer display 
that monitors one of the user's eyes. The device also 
contains an infrared light emitting diode (LED) located 
at the center of the camera's lens to maximize the bright- 
eye affect. Image processing software on the computer 
computes the user's gazepointon the display sixty times 
a second with an accuracy of about a quarter inch. 

Gaze tracking devices have been used for weapon 
control, operator training, usability analysis, market re- 
search, and as an enablement for the disabled. Howev- 
er, gaze tracking devices have not been used to deter- 
mine what characteristic of a computer's operation in- 
terests the computer user at a particular time or to allow 
the computer to adapt to a user's interest as demonstrat- 
ed by where on the display screen the user is looking. 

Text to Speech 

Many modern computers now provide text-to- 
speech capability. This capability processes text strings 
and produces understandable audio speech from the 
computer's audio output device (headphones or speak- 
er). This capability allows a computer to present an au- 
dio version of a text string to a computer user. 

Problems with Downloading Information 

The background of the World Wide Web (WWW) 
and WWW browsers are well described by reference to 
the first chapter of instant HTML Web Pages, by Wayne 
Ause, Ziff-Davis Press, ISBN 1-56276-363-6, Copyright 
1995, pages 1-15, hereby incorporated by reference as 
illustrative of the prior art. 

Using the Internet, a computer user has access to 
an immense amount of information. However, retrieving 
this information over the Internet often takes significant 
time because of the limited bandwidth of the communi- 
cation channel. The bandwidth is limited by many fac- 


tors. Some of these factors are the bandwidth of the 
communication link from the user's computer to the In- 
ternet, the bandwidth of the communication link from the 
information provider's computer to the Internet, the ex- 
5 istence of other communication traffic on these links, 
and the bandwidth of the Internet itself. Often, the pri- 
mary bandwidth limitation is at the user's computer. 

This bandwidth limitation at the user's computer is 
exacerbated because multiple data streams often flow 
across this limited communication link. If the user is in- 
terested in a particular data transfer, these additional da- 
ta streams utilize bandwidth that would otherwise be 
available to the data stream-of-interest to the user. This 
results in a decreased data transfer rate of the data 
stream-of-interest. 

Prior art WWW browsers, for example, generally at- 
tempt to equally allocate bandwidth to all the data trans- 
fers directed towards visible views in a window. Al- 
though this approach is clearly better then simply se- 
quentially retrieving data for each view, this approach 
delays retrieving data that is of the most interest to the 
user because the available channel bandwidth is divid- 
ed between the data streams supplying data to the 
views. Thus, the user must wait an additional time be- 
cause of uninteresting information using bandwidth that 
could have been applied to the information of interest. 

During the transmission of large amounts of data, a 
program generally provides some indication of the 
progress of the transmission. This indication is provided 
by indicators such as bar indicators, numerical percent- 
age indicators, or in the case of images often just the 
amount of detail available in the displayed image. While 
waiting for the transfer to complete, the user often 
watches the progress of the indicator or of the partially- 
filled image. 

As mentioned above, one problem with the prior art 
is that a user has little control over the bandwidth allo- 
cated to the data stream used to download information. 
Further even if an application should provide this control 
to the user, the user still must explicitly command the 
application to set the allocated bandwidth. 
The invention addresses these problems. 

Problems with Additional Data Associated with 
Images 

In print and computer hypertext documents, images 
such as pictures and illustrations, are often provided 
with additional information, such as captions explaining 
or enhancing the image. Those who viewthe image can- 
not look at the image and read an associated caption at 
the same time. Thus, the viewer's attention is diverted 
from the image while searching for, and reading, the as- 
sociated caption. Contrast this situation with a directive 
time-dependent medium, such as film or video, where 
a viewer is simultaneously presented with both visual 
and audio information. Audio captioning presents addi- 
tional information though an audio speaker allowing the 
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user to receive additional information auditorally without 
distracting the viewer's gaze from the image of interest. 
Systems that allow a user to select which image to view, 
from a plurality of images, require the user to explicitly 
trigger the vocal caption. Thus, the user is again dis- 
tracted from looking at the image by the need to seek 
out and activate the caption. 

The invention addresses these problems. 

Problems with Small Text Displayed to a User 

People often have difficulty reading text on a com- 
puter display screen. Often this is due to vision difficul- 
ties. Thus, the type used in WYSIWYG (what you see 
is what you get) applications is often too small for com- 
fortable reading at the display distance. Further, pub- 
lishers use different type sizes as a layout tool that indi- 
cates importance. Thus, there is a large variation in text 
size and screen space used between the largest head- 
line text and the text of an article. To address this prob- 
lem, some applications allow the WYSIWYG text to be 
magnified. Examples of word processing programs that 
provide this capability are Microsoft's Word® and Ado- 
be's FrameMaker® programs. However, these pro- 
grams require the user to explicitly specify, either direct- 
ly or indirectly, the desired magnification factor. Further, 
the magnification process reduces the amount of the 
page that can be displayed on the computer display at 
the same time because the percentage of the page that 
is displayed to the user is reduced when the page is 
magnified. This problem is exacerbated with applica- 
tions that display WYSIWYG versions of newspapers 
and magazines because these applications generally 
attempt to maintain the WYSIWYG page layout and the 
displayed page is uniformly magnified. To see the entire 
page, the article text is generally reduced to unreadable 
size. 

The page layout of newspapers and magazines is 
important. To attract the interest of a large number of 
readers, the publishers of newspapers present a large 
number of articles on the first few pages. One way to 
increase the number of articles on a page is to decrease 
the amount of space used for the article. In a traditional 
newspaper, this is accomplished by moving subsequent 
parts of the article to different pages. This allows the 
reader to quickly scan articles that the editor believes to 
be most relevant and to read in depth those articles that 
the reader finds interesting. Further, where articles are 
placed on a page influences the order that articles are 
viewed. Electronic newspapers have these same char- 
acteristics. 

Additionally, electronic newspapers, like traditional 
newspapers, use different type styles and sizes to indi- 
cate the relative importance of headlines and subhead- 
ers. Thus, there is a wide disparity between the largest 
and smallest text displayed to the reader. Moreover, 
even large computer displays have a smaller display ar- 
ea than is available to a traditional newspaper thus re- 


ducing the area available to the publisher for articles. 

Nevertheless, a computer display must often carry 
the same amount of information as a newspaper. Thus, 
mapping the content of a newspaper onto a display 
5 screen reduces the size of the type used for the articles 
to the point where the text of the article is extremely dif- 
ficult to read. Further, the magnification method used by 
word processing programs for globally expanding the 
displayed text does not work well when presenting many 
articles on a page because magnifying the entire page, 
and providing a limited view into the page distances the 
structure of the page from the viewer. Thus, globally ex- 
panding the text is incompatible with presenting as 
many articles as is desired on the page. Further, globally 
expanding the page also expands the larger title and 
headline text more than is needed to make this text read- 
able and at a cost of consuming undue display space 
that could otherwise be used to present additional infor- 
mation. Thus, there is a need for a mechanism that op- 
timizes the text size for a reader while still preserving 
the structural indications provided by the page layout. 
The invention addresses these problems. 

Problems with Selecting Relevant Information for a 
User 

Another aspect of electronic newspapers, briefly 
mentioned above, is that of selecting information con- 
tent for the newspaper. Information content includes 
both articles about particular items of interest and ad- 
vertising information, information content is a major rea- 
son why people select different paper-based magazines 
and newspapers. These traditional information provid- 
ers present the information that they believe interest 
their readers. Traditional newspapers and magazines 
are static once printed. Thus, each edition is the same 
for all those who read it and each copy of a particular 
edition distributed to a particular region has the same 
articles : the same layout and the same advertising as 
all the other copies distributed to the region. This adver- 
tising and information content can be customized to the 
particular region. However, this regionalization can only 
be carried so far as it is extremely expensive to custom- 
ize a newspaper or magazine to the particular individual 
interests of each reader. Thus, some of the information 
selected for a region will not interest some readers. 

Intangible electronic newspapers need not be con- 
strained by the above mentioned limitations inherent in 
using a tangible paper medium. However, electronic 
newspapers still target most advertising and information 
content to a particular market and not to the particular 
interests of an individual reader. Even where the reader 
of an electronic publication is provided with a means to 
customize the content of the electronic paper the user 
must explicitly specify the content. Further, by explicitly 
specifying the content, the user may not be presented 
with other related information that falls just outside of 
the specification but that could be of interest to the read- 
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The invention addresses these problems. 

Summary at the Invention 

The present invention provides an economical, ap- 
paratus, method, system and computer program prod- 
uct for providing enhanced facilities to computer users. 
This invention provides a way for a computer to monitor 
the user to determine what aspect of the computer op- 
eration the user is interested in and to respond accord- 
ingly. 

One aspect of the invention is a computer controlled 
method for displaying a portion of some of a plurality of 
articles to a computer user. The computer is equipped 
with a display device and a gaze-tracking device. The 
gaze-tracking device being capable of determining a 
gaze position on said display device. The computer con- 
trolled method displays portions of articles on the dis- 
play device. Next the method detects an article of inter- 
est by determining an intersection between the gaze po- 
sition and a portion of one of the articles. Finally, the 
method expands the displayed portion of the article of 
interest on the display. 

In another aspect of the invention, an information 
presentation apparatus is disclosed having a central 
processing unit, memory a display device and a gaze 
tracking device capable of determining a gaze position 
on the display device. The apparatus includes a display 
mechanism than is configured to display portions of a 
plurality of articles on the display device. The apparatus 
also includes a detection mechanism that detects an in- 
tersection of a portion of one of the displayed articles 
with the user's gaze and thus determines an article of 
interest to the user. Finally, the apparatus includes an 
expansion mechanism that expands the portion of the 
article of interest on the display device. 

Yet another aspect of the invention discloses an in- 
formation presentation system. The system includes a 
gaze-tracking device that determines a gaze position on 
the display device where the user is looking. The system 
includes a display mechanism that displays portions of 
articles on a display device. Additionally, the system in- 
cludes a detection mechanism that determines an arti- 
cle of interest by detecting an intersection between the 
gaze position and a portion of a displayed article. Finally, 
the system includes an expansion mechanism that ex- 
pands the displayed portion of the article of interest. 

A final aspect of the invention discloses a computer 
program product having computer readable code em- 
bodied in a computer usable storage medium. The com- 
puter readable code causes a computer to effect a gaze 
position determination mechanism used to determine a 
gaze position from gaze coordinates returned from a 
gaze-tracking device. Further the invention includes 
code that displays portions of a number of articles on a 
display device and detects when the gaze position in- 
tersects one of these portions. This article containing 
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that intersected portion is the article of interest. Finally, 
the invention includes code that expands the intersected 
portion of the article of interest. 

The foregoing and many other objects and advan- 
tages of the present invention will no doubt become ob- 
vious to those of ordinary skill in the art after having read 
the following detailed description of the preferred em- 
bodiments which are illustrated in the various drawing 
figures. 

Description of the Drawings 


Figure 1 


Figure 2 
20 Figure 3 

Figure 4 

25 

Figure 5 


Figure 6 

Figures 7a & b 
Figure 8 

Figures 9 

Figure 10 
Figure 1 1 

Figure 12 

Figure 13 

Figure 14 


illustrates a portion of a computer 
system, including a CPU and a con- 
ventional memory in which the 
present invention may be embod- 
ied; 

illustrates a display device fitted 
with gaze tracking equipment; 
illustrates aspects of a gaze posi- 
tion in accordance with a preferred 
embodiment; 

illustrates the process for determin- 
ing a gaze position as used in ac- 
cordance with a preferred embodi- 
ment; 

illustrates the operation of the in- 
vention to allocate bandwidth to an 
area of interest in accordance with 
a preferred embodiment; 
illustrates the process used to 
change the bandwidth of a data 
stream based upon a gaze position 
in accordance with a preferred em- 
bodiment; 

illustrate audio captioning in ac- 
cordance with a preferred embodi- 
ment; 

illustrates extraction of an image 
caption from a page of text in ac- 
cordance with a preferred embodi- 
ment; 

illustrates the process used to im- 
plement captioning in accordance 
with a preferred embodiment; 
illustrates the form of an electronic 
newspaper; 

illustrates text magnification and 
page layout in accordance with a 
preferred embodiment; 
illustrates text magnification and 
page layout in accordance with a 
second preferred embodiment; 
illustrates the process of expanding 
text in response to the users inter- 
est in the text in accordance with a 
preferred embodiment; 
illustrates the process for adjusting 
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the layout of a display as a result of 
expanded text in accordance with a 
preferred embodiment; 
Figure 1 5 illustrates a possible first page of an 

electronic newspaper showing arti- 5 
cles and an advertisement in ac- 
cordance with a preferred embodi- 
ment; 

Figure 16 illustrates a possible second page 

of an electronic newspaper show- 10 
ing information determined to be of 
interest to the reader in accordance 
with a preferred embodiment; and 

Figure 17 illustrates the process used to eval- 

uate the information of interest to a 15 
reader and to select new informa- 
tion matching the readers interest in 
accordance with a preferred em- 
bodiment. 

20 

Description of the Preferred Embodiments 


E-mail system - Electronic mail system. A system 
of computers generally connected by a networkthat 
allow a sender (being a user of a first computer) to 
compose and send data making up a message to a 
recipient (being a user of either the first computer 
or of a second computer). 

Graphical User Interface (GUI) - A user interface 
that allows a user to interact with a computer display 
by pointing at selectable control areas on the dis- 
play and activating a command or computer oper- 
ation associated with the selectable control area. 
GUIs are well known in the art. 

Gaze position - An area of interest on the screen 
providing a boundary of the user's gaze over a lim- 
ited period of time. 

Gaze coordinates - The coordinates that represent 
the intersection of the user's gaze with the display 
screen over a limited period of time. 


Notations and Nomenclature 

The following "notations and nomenclature" are 
provided to assist in the understanding of the present 
invention and the preferred embodiments thereof. 

Advertisement - Information provided about a 
commercial product or service with the purpose of 
informing the viewer about the product or service 
so as to lead to a commercial transaction. A type of 
article. 


Gaze coordinates (raw) - The coordinates that 
represent the instantaneous intersection of the us- 
25 er's gaze with the display screen. 

Image - Any information displayed on a display 
screen such as, but not limited to, pictures, draw- 
ings, illustrations, text, and video. An image gener- 
30 ally displayed in a view contained in a window. A 
still image is a picture. A moving image is comprised 
of a number of frames of still images that are played 
in sequence similar to a video 


Article - A complete piece of writing often identified 35 
with a title. 

Bandwidth - The amount of information that can be 
passed across a communication channel in a given 
period of time (usually designated in reference to a 40 
second). 

Dialog - A specialized window that is used to obtain 
additional information from the user. A dialog is of- 
ten used to obtain options and parameters that are 45 
computer dependent. A good example is a print di- 
alog that is evoked by a print menu command. The 
print dialog allows the user to specify what printer 
options are to be used for a particular print job. Gen- 
erally the dialog allows the user to specify specific 50 
parameters and then to either affirm or cancel the 
command that evoked the dialog. If the user cancels 
the command, the dialog window is removed and 
the command that evoked the dialog is aborted. If 
the user confirms the command the user provided 55 
information acquired by the dialog is used in the ex- 
ecution of the command that evoked the dialog. 


Pointing device - A device responsive to a compu- 
ter user's input that moves an indicator on a com- 
puter display screen. Such an indicator has an ac- 
tive point such that if the pointing device is activated 
(e.g., by a button push for a mouse device) a com- 
mand associated with the selectable control area 
covered by the active point is invoked. Pointing de- 
vices are generally used with graphical user inter- 
faces. 

Selectable control area - An area on a computer 
display that is sensitive to activation of a pointing 
device. On activation of the pointing device over the 
selectable control area, a command or computer 
operation associated with the selectable control ar- 
ea is invoked. Most computer systems that provide 
a Graphical User Interface (GUI) also provide other 
methods for invoking these commands or computer 
operations such as keyboard function keys or com- 
mand lines. 

URL - A Uniform Resource Locator. URLs are used 
to access information on the World Wide Web. 
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View - An area in a window where information is 
provided. 

Window - An area ; usually rectangular, on a com- 
puter display screen controlled by an application. 

Procedure - A self-consistent sequence of steps 
leading to a desired result. These steps are those 
requiring physical manipulation of physical quanti- 
ties. Usually these quantities take the form of elec- 
trical or magnetic signals capable of being stored, 
transferred, combined, compared, and otherwise 
manipulated. These signals are referred to as bits, 
values, elements, symbols, characters, terms, 
numbers, or the like. It will be understood by those 
skilled in the art that all of these and similar terms 
are associated with the appropriate physical quan- 
tities and are merely convenient labels applied to 
these quantities. 

Overview 

The manipulations performed by a computer in ex- 
ecuting opcodes are often referred to in terms, such as 
adding or comparing, that are commonly associated 
with mental operations performed by a human operator. 
In the present invention no such capability of a human 
operator is necessary in any of the operations described 
herein. The operations are machine operations. Useful 
machines for performing the operations of the invention 
include programmed general purpose digital computers 
or similar devices. In all cases the method of computa- 
tion is distinguished from the method of operation in op- 
erating a computer. The present invention relates to 
method steps for operating a computer in processing 
electrical or other (e.g., mechanical, chemical) physical 
signals to generate other desired physical signals. 

The invention also relates to apparatus for perform- 
ing these operations. This apparatus may be specially 
constructed for the required purposes or it may com- 
prise a general purpose computer as selectively activat- 
ed or reconfigured by a computer program stored in the 
memory of a computer. The procedures presented here- 
in are not inherently related to a particular computer or 
other apparatus. In particular, various general purpose 
machines may be used with programs written in accord- 
ance with the teachings herein, or it may prove more 
convenient to construct more specialized apparatus to 
perform the required method steps. The required struc- 
ture for a variety of these machines will appear from the 
following description. Also, the invention may be em- 
bodied in a computer readable storage medium encod- 
ed with a program that causes a computer to perform 
the programmed logic. 

Operating Environment 

Figure 1 illustrates a computer system referenced 


to by the general reference character 1 02, configured to 
support the invention. The system 102 includes a proc- 
essor 133 having an Input/Output ("I/O") section 135, a 
central processing unit ("CPU") 137 and a memory sec- 

s tion 1 39. The I/O section 1 35 is connected to a keyboard 
141, a disk storage unit 143, a network interface 145 to 
provide access to a network 117, a display unit 147, a 
pointing device 148, a gaze-tracker device 155, a 
speaker 157 and a CD-ROM drive unit 149. The CD- 

10 ROM unit 149 can read a CD-ROM medium 151 that 
typically contains a plurality of programs 153 and data. 
The CD-ROM 149 and the disk storage unit 143 com- 
prising a filestorage mechanism. One skilled in the art 
will understand that the filestorage mechanism may 

15 comprise read only memory, RAM or other storage tech- 
nology that allows a computer to access data. Such a 
computer system is capable of executing programmed 
logic that embodies the invention. 

20 Gaze Tracker Operation 

Figure 2 illustrates a gaze tracking device 201 at- 
tached to a computer display device 203. As illustrated, 
the gaze tracking device 201 is mounted below the com- 

25 puter display 203 and comprises a video lens 205, with 
an infrared LED 207 mounted on the center of the lens 
205. One skilled in the art will understand that one of 
many different types of gaze tracking devices can be 
used with the invention. The computer display device 

30 203 has a display area 209 that the computer can ac- 
cess to display information. The computer generally cre- 
ates a plurality of windows 211 and 213 for this informa- 
tional display. As the user looks at the display area 209, 
the gaze tracking device determines the users gaze po- 

35 sition. The gaze coordinate (raw) is an ordered pair of 
values providing the immediate two dimensional coor- 
dinates of where the user's gaze is positioned on the 
screen. A number of gaze coordinates can be used to 
develop a gaze position that is a two dimensional coor- 

40 dinate of a time weighted average of the gaze coordi- 
nates. The invention need not have precise gaze coor- 
dinates. In fact the gaze coordinates need only be ac- 
curate to approximately a 1/4 inch. One skilled in the art 
will understand that the gaze position could also be re- 

45 turned as a rectangle, polygon, circle or other shape rep- 
resenting an area of interest. Further one skilled in the 
art will understand that the gaze tracking device 201 
may be configured to provide the gaze position only 
when the user has shown interest in a displayed area 

50 for a sufficient time, or that application or system pro- 
gram logic may be utilized to detect when the user has 
shown an interest by monitoring the gaze coordinates 
over some period of time. The gaze position is devel- 
oped when the system detects that the user has become 

55 interested in a particular area of the display. The system 
detects this condition by recognizing that the user's 
gaze has been limited to a particular region of the dis- 
playable area 209 for a period of time. 
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Figure 3 illustrates aspects of a gaze position de- 
termined from gaze coordinates returned from a gaze 
tracking device. The dashed rectangle labeled as 301 
indicates the boundaries of an image (not shown). Be- 
cause the gaze tracking device has an uncertainty and 
because the user's gaze constantly moves (because of 
involuntary eye movements) even when focused on an 
area of interest, this embodiment of the invention pro- 
vides the gaze position as an area 305 where the user's 
gaze is fixed. The provided area 305 indicates that the 
user's gaze did not leave the specified area for a given 
time such as a half second. The application program that 
uses the returned circular area 305 is able to determine 
that the user's gaze is well within the image 301 because 
the provided circle is completely contained within the ar- 
ea of the image. If the user has interest in the entire im- 
age, the gaze would be more diffused resulting in a larg- 
er circle. For example, in comparing the image labeled 
301 with an image 31 1 , the user viewing the image 301 
has focused in on a particular aspect of the image as 
indicated by the circle labeled as 305. The user viewing 
the image 311 has indicated more interest in the totality 
of the image as compared to a particular aspect of the 
image as indicated by the circle labeled as 309. One 
skilled in the art will understand that a rectangle or other 
area indication can be used as well as a circle. 

In another embodiment of the invention, the gaze 
tracker returns the gaze position as a point. For exam- 
ple, a dashed rectangle labeled as 303 indicates the 
boundaries of a different image (again this image is not 
shown). In this embodiment, the gaze tracker, after 
processing gaze movements over some period of time, 
simply returns a point that is a time weighted average 
of the position of the gaze. This point is indicated by the 
arrow labeled as 307. 

Figure 4 illustrates the process used to develop a 
gaze position from a plurality of gaze coordinates. The 
process starts at a terminal 401. Next at a step 403, the 
process performs any necessary setup or initialization. 
This includes initializing timer program logic to periodi- 
cally gather instantaneous gaze coordinate data from 
the gaze tracker apparatus, acquiring user preference 
data and other one-time initializations. Next at a step 
405, the process receives a plurality of gaze coordinates 
(raw). These gaze coordinates (raw) indicate the posi- 
tion of the user's gaze over a given time. After a suffi- 
cient number of gaze coordinates (raw) are gathered, 
the process examines the gaze coordinates (raw) for a 
focus pattern at a branch point 407. A focus pattern oc- 
curs when the gaze coordinates (raw) are all within 
some boundary without gaze coordinates (raw) being 
statistically far from the others. If the gaze coordinates 
(raw) do not show a focused gaze, the gathering of in- 
stantaneous gaze coordinates (raw) continues as 
shown by an arrow 406. If at the branch point 407 a focus 
pattern is found then a step 409 determines an area of 
focus that contains the statistically significant gaze co- 
ordinates (raw). Finally, in a step 411 the gaze position 


is made available to the program logic for use. Finally, 
the process repeats as shown by an arrow 412. In the 
case where the gaze position is presented as an area 
instead of a simple coordinate pair, the receiving pro- 

5 gram logic determines which view is overlapped by the 
gaze position. 

There are various well known methods in the art to 
present the gaze position to program logic such as an 
application program. These include providing an excep- 

10 tion to an application, sending an interprogram message 
containing the gaze position to the application and many 
other methods. 

An executing computer program can modify its op- 
eration to best suit the user's interest by using a gaze 

15 tracking device to determine which area of a display 
screen is the object of a user's attention. Specific em- 
bodiments using this capability of the gaze tracker are 
described below. 


As mentioned above, prior art communication pro- 
grams such as WWW browsers and other data transfer 
programs that provide multiple data streams do not al- 

25 locate bandwidth depending on the user's specific inter- 
est. At best these programs will allocate bandwidth to 
images and data that is presented to the user on the 
display device. The invention enables a computer to de- 
tect which data stream the user is interested in and to 

30 allocate more bandwidth to that data stream without an 
explicit command from the user. 

Figure 5 illustrates how a preferred embodiment of 
the invention is used in a WWW browser. The browser 
application displays a window 501 on the display device. 

35 The user invokes a URL to present a webpage contain- 
ing information encoded in HTML in the window 501. In 
this particular example, the webpage shows a plurality 
of three areas of text 503, 511 and 517 along with a plu- 
rality of areas of images 505, 51 3 and 519. Each of these 

40 images are displayed in a view. Further, these images 
are constructed from "large data entities". Each large 
date entity is composed of a large amount of data that 
is used to define the entity. Examples of large data en- 
tities, among others, are high resolution graphical imag- 

45 es, sound and video entities. A plurality of shaded areas 
507, 509 and 521 of the images 505, 513 and 519 indi- 
cate the amount of the image that has been downloaded 
at some particular point in time prior to completion. If the 
user is interested in the image labeled as 513, the user 

50 will watch that image 513 fill. Thus, a gaze position area 
515 intersects the image 513 thus identifying the image 
513 as an area of interest. A complementary approach 
to determining which image 505, 51 3 or 519 is of interest 
is to have the text 503, 511 or 517 associated with each 

55 image. Thus, when the user reads the text 511 linked 
(associated) with an image 513 a gaze position 512 is 
found on the text. Once the user shows interest in the 
text, the application can properly assume that the area 
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of interest is the linked image 513. 

Once the area of interest is determined, the system 
obtains the bandwidth allocated to the area of interest 
and, if possible, increases the bandwidth allocation of 
the data stream directed to the area of interest using 
techniques well understood in the art. 

Figure 6 illustrates the process used to allocate 
bandwidth to a data stream supplying data to an area of 
interest. The process starts at a terminal 601. At a step 
603 the process receives the gaze position from a gaze 
tracking device as described above. Then at a step 605 
the gaze position is used to determine what information 
or view on the display device intersects with the gaze 
position. Next in a decision block step 607, the process 
checks whether the displayed information that inter- 
sects the gaze position is relevant. Information that is 
relevant comprises information that is in the process of 
being transferred to the display device. In other words, 
a completely loaded image is not relevant and has no 
bandwidth allocation because there is no need to in- 
crease the bandwidth for an image that has been com- 
pletely transmitted. Contrast this situation where there 
is zero bandwidth allocated to a yet-to-be-loaded image. 
Here the yet-to-be-loaded image has a bandwidth allo- 
cation, but the current allocation is zero whereas there 
is no bandwidth allocation at all for an image that has 
been completely downloaded. Text is another example 
of displayed information that usually is not relevant be- 
cause the user cannot read the text as fast as it is trans- 
mitted even with limited bandwidth. 

If in the decision block step 607 the displayed infor- 
mation is relevant then, in a step 609 the process deter- 
mines the area of interest on the display and in a step 
611 increases the bandwidth allocated to the data flow 
associated with the area of interest. Finally, the process 
completes through a terminal 613. 

However, if in the decision block step 607 the dis- 
played information is not relevant, the source of the dis- 
played information is checked in a decision block step 
615 to determine whether the source indicates that the 
displayed information is associated with an area of in- 
terest. An example of this situation is where text is as- 
sociated with an image. If at the decision block step 615 
the source is not associated with an area of interest 
processing completes through the terminal labeled as 
613. However, if 615 the source is associated with an 
area of interest, the process 611 increases the band- 
width of data flowing to that area of interest. Finally the 
process completes through the terminal 613. 

In a preferred embodiment of a WWW browser the 
process illustrated in Figure 6 determines what area of 
the display device the user is interested in. If that area 
is a partially downloaded image, the invention increases 
the bandwidth allocated to the data flow for completing 
the image. If the area is a textual area whose defining 
source HTML provides an association with an image (for 
example, by providing an attribute within the paragraph 
element that associates a large data entity with the par- 


agraph) the invention increases the bandwidth allocated 
to the data flow for completing that specific image. 

Finally, embodiments utilizing the invention are not 
limited to WWW browsers. Any application that presents 
5 information that is downloaded across a limited band- 
width network onto a display device can use the inven- 
tion. This includes detecting interest on data transfer 
status displays generated by programs such as, among 
others : FTP Kermit, or x- y- and zmodem applications 

10 

Captioning 

Another preferred embodiment of the invention re- 
lates to captioning of images. Audio captioning detects 
15 the user's interest in a displayed image and automati- 
cally provides additional audio information to the user 
about the displayed image while video captioning pro- 
vides additional video information. 

Figure 7a illustrates captioning. An image of a 
20 phone 701 is displayed to the user. If the user is inter- 
ested in the phone, the user looks at the image 701 thus 
generating a gaze position from the gaze tracking de- 
vice. The invention then presents additional information 
about the phone to the user. In some circumstances, the 
25 additional information is electronic information attached 
to the image by a data structure within the underlying 
application such as the one shown in Figure 7b. Such 
additional information may include text, text that is con- 
verted to audio, recorded audio data, video data, or an- 
30 other image. 

In other circumstances the additional information is 
a textual caption 703 displayed with the image 701 but 
without a direct electronic linkage to the captioned im- 
age 701 . When the additional information is a displayed 
35 textual caption 703, the invention detects that a textual 
caption 703 exists for the image 701, extracts that tex- 
tual caption 703 and passes the extracted text string to 
a text-to-speech process that presents the caption text 
to the user in audio form. Some page markup languages 
40 support a <CAPTION> tag. For example, both SGML 
and HTML support such a tag (note however, that al- 
though HTML supports a <CAPTION> tag, it is currently 
only used for table captions and not for figures). Thus 
one of the preferred embodiments of the invention uses 
45 the <CAPTION> tag to detect the textual caption 703 of 
an image 701. 

All of these methods provide additional information 
to the user about an image-of-interest. The audio meth- 
ods allow the user to maintain his/her attention on the 
50 image while receiving additional information auditorally. 
Many of the other methods of presenting the additional 
information result in the user removing her/his gaze from 
the image-of-interest. 

Figure 7b illustrates a representative data structure 
55 710 that can be used to associate additional electronic 
information with an image as mentioned above. An "Im- 
age ID" field 711 identifies the image displayed to the 
user. A "Pointer to Image Data" field 713 contains the 
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image data, or data that directly or indirectly references 
the image data used to create the displayed image 701 . 
A "Pointer to Additional Information" field 715 contains 
the additional electronic information data, or data that 
directly or indirectly references the additional electronic 
information data. This data structure is one of many that 
can be used by programmed applications that provide 
an explicit link between an image and additional infor- 
mation. Some programmed applications do not provide 
such an explicit linkage. 

Figure 8 illustrates a window 801 displaying an ex- 
ample page 803. The page 803 includes text and an im- 
age 805 of a phone. The phone image 805 has an as- 
sociated textual caption 807. The image 805 and the 
caption 807 are offset from the text by a plurality of blank 
areas 809, 811 and 813. These blank areas 809, 811, 
813 delimit the image 805 and its associated textual 
caption 807 from the rest of the displayed text. This sep- 
aration allows the underlying application program to de- 
termine a linkage between the image and its associated 
textual caption. Further, even if the underlying pro- 
grammed application that presented the window 801 
does not have the facility to directly determine the link 
between the caption 807 and the image 805, other pro- 
grammed applications can scan the window 801. After 
scarring the window these programmed applications 
then isolate and pass the caption text 807 through a text- 
to-speech facility to convert the caption text 807 to audio 
and present the audio to the user. 

Figure 9 illustrates the process used to present ad- 
ditional information. The process starts at a terminal 951 
when it receives a gaze position from a gaze-tracking 
device at a step 953. If at a decision block step 955 the 
gaze position does not intersect an image, the process 
completes through a terminal 957. If at the decision 
block step 955 the gaze position intersects an image 
and if at a decision block step 959 the image has an 
electronic caption, the process accesses and presents 
the additional information at a step 961 . Finally, the proc- 
ess completes through the terminal 957. If at the deci- 
sion block step 959 the image does not have an elec- 
tronic caption but does have a displayed textual caption 
as determined in a decision block step 963, the process 
extracts the caption text and passes the resulting text 
string to a text-to-speech facility for auditory presenta- 
tion to the user in a step 965. Finally, the process com- 
pletes through the terminal 957. If in the decision block 
step 963 the image does not have a textual caption, or 
if such a caption cannot be determined, the process 
completes through the terminal 957. 

One skilled in the art will understand that auditory 
presentation of the additional information to the user is 
accomplished with techniques known in the prior art. 

Displayed Text Enlargement 

A page from an electronic newspaper is illustrated 
in Figure 10. This information is displayed on a compu- 


ter display screen generally in a window as discussed 
above. Here a window 1001 contains a headline 1003, 
a plurality of four article titles 1005, 1009, 101 3 and 1017 
and a partial text 1 007, 1 01 1 , 1 01 5, and 1 01 9 of the titled 

5 articles. A computer for displaying the electronic news- 
paper that is equipped with a gaze tracking device can 
rely on the normal reading pattern of a user. That is, that 
the user first reads the major headline 1003, then the 
article title 1005, 1009, 1013 or 1017 followed by the 

10 article itself 1007, 1011, 1015, or 1019. Thus, when the 
gaze tracker indicates that the user is reading an article 
title, the invention starts expanding the article text by a 
magnification factor to an optimal size for the user. 
Figure 11 illustrates two possible embodiments of 

15 the invention. As in Figure 10 within a window 1101, a 
plurality of four article titles 1105, 1109,1113, 1117 are 
displayed. The window 1101 also includes a headline 
1103. An article 1107 has been expanded by the inven- 
tion to a comfortable size for reading. This expansion 

20 has occurred without increasing the size of the window 
1101 within which the article 1107 is displayed. Howev- 
er the view containing the article 1107 and the text of 
the article 11 07 have been expanded. To obtain the nec- 
essary window space to expand the view containing the 

25 article 1107 without interfering with the expanded view 
1107, the view containing an article 1115 along with its 
title 1 1 1 3 have been reduced in size and shifted towards 
the edge of the window 1101. The articles associated 
with the titles 1109 and 1117 have been shifted com- 

30 pletely off of the window 1101 leaving only their associ- 
ated titles 1109 and 1117. 

If the computer detects that the user's gaze has 
moved from the expanded article 1 107, to a different title 
1113, 1109, or 1117, the text of the associated article is 

35 expanded. At the same time, the text and title of the pre- 
viously expanded article 1105, 1107 is slowly reduced 
so that the user's gaze is not attracted by the apparent 
movement caused by the reductionln a preferred em- 
bodiment of the invention the user may utilize computer 

40 commands to cause the reduction and expansion of the 
articles and their associated title to be instantaneous. 
Further the expansion can be to a predetermined user 
preference. Finally, the expansion of the article can be 
dynamically determined by monitoring the user's read- 

45 jng speed of the article (by determining how the gaze 
position changes over time) so as to maximize the read- 
ing speed. 

Another embodiment of the invention retains the 
original size of the view, but magnifies the text within the 

50 view by a magnification factor. This embodiment does 
not require the reduction and repositioning of the other 
articles as does the preferred embodiment However, as 
the text expands within the limited bounds of the view 
the amount of text that can be displayed in the unmag- 

55 nified view is reduced. Further, the maximum magnifi- 
cation of the text is limited in that the essential structure 
of the left to right reading pattern (for Latin based lan- 
guages) should be retained for comfortable reading. Too 
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much magnification of the text within a limited space re- 
sults in a column of words that is difficult to read. 

Figure 12 illustrates another approach to managing 
the information displayed in a window 1 201 . Here, a title 
1205 and text of an article AA 1207 is expanded and 
simply overlays a plurality of other articles 1211, 1215, 
and 1219 in the window 1201 . Because the other articles 
and titles are obscured, the text is reduced when the 
user's gaze leaves the window 1201 or when the user 
invokes a command function either by use of a pointing 
device, a command line or other command invocation 
method. Because the magnified view overlays the other 
views, a preferred embodiment places the magnified 
view within a boarder 1217 to separate the magnified 
view from the other views. 

Figure 1 3 illustrates the process used to determine 
when to expand an article. The following description ref- 
erences Figure 1 0 as an illustrative example. The proc- 
ess starts at a terminal 1 301 . At this time the user's gaze 
has not focused in on a window 1001. When the user's 
gaze enters the window 1001, the process, at a step 

1305, monitors which text is read by the user. So long 
as the gaze position indicates that the user is reading 
one of the plurality of titles 1005, 1009, 1013, and 1017 
as determined by a decision block step 1307 the moni- 
toring step 1305 continues as indicated by an arrow 

1306. If the decision block step 1307 detects that the 
user's gaze changedtoasubtitleorthe plurality of article 
text 1007, 1011, 1015, 1019, the process starts to ex- 
pand the view and text of the contained article at a step 
1309. The process continues to monitor the user's gaze 
position at a step 1311 to determine the user's reading 
speed. Next at a step 1315, the process adjusts the size 
of the article text to maximize the user's reading speed 
possibly continuing the expansion of the view started in 
the step 1311. This adjustment uses a magnification fac- 
tor having a value that is a function of the reading speed. 
This magnification factor controls the view and the text 
within the view. Finally at a decision block 1317, the 
process determines whether the user's gaze position 
continues to intersect the view containing the article 
(that is, to verify that the user continues to read the ar- 
ticle). If at the decision block 1317 the user continues to 
read the article, the monitoring process of the step 1311 
continues as indicated by an arrow 1313. However if at 
the decision block 1317, the user has stopped reading 
the article, the process starts reducing the size of the 
abandoned article at a step 1319 and the entire process 
is repeated as indicated by an arrow 1 321 . The invention 
allows the user to reacquire the abandoned article sim- 
ply by bringing the user's gaze back to the text of the 
abandoned article. This stops the reduction of the size 
of the article text and restarts the expansion of the article 
text at the step 1309 and leading to the text adjustment 
step 1315. 

In the enlarging and reduction steps above, the text 
is slowly enlarged and reduced so as not to be a distrac- 
tion to the user. Thus the change in size does not attract 


the user's gaze from the article being read. However, 
the invention also provides the user with commands that 
accelerate these scaling processes. Further, the maxi- 
mum size of an expanded article can be specified as a 
5 user preference. 

Figure 14 illustrates the process used to size the 
views impacted by the magnification of the view contain- 
ing the article being read by the user - the active view. 
The process starts at a terminal 1401. At a step 1403 
the process first determines the new position and size 
of the active view in response to the change of scale. 
Based on this new position and size, at a step 1405 the 
process determines which other views displayed in the 
window 1001 will be affected by the change in size and 
position of the active view. Then, at a step 1407, each 
affected view is determined and adjusted in a step 1409 
to make room for the enlarged active view (where the 
active view is expanding), or to take up room released 
by a reduced active view (where the active view is con- 
tracting). This process is repeated as indicated by an 
arrow 1411 for each impacted view. Once the impacted 
views are adjusted the process continues, as indicated 
by an arrow 1415, to adjust the size and position of the 
active view in a step 1417. Finally, the process com- 
pletes through a terminal 1419. Note that a reduction in 
the size of the active view occurs to optimize the user's 
reading speed. This is not the same reduction that re- 
sults when the user has stopped reading a view and that 
view contracts because it is no longer the active view. 

This aspect of the invention has been described as- 
suming normal English reading patterns (that is, top to 
bottom, left to right). One skilled in the art will under- 
stand that other reading patterns are contemplated by 
the invention. 

Interest Based Information Presentation 

Figure 15 illustrates a typical electronic newspaper 
display. The invention also applies to data search en- 
gine displays. Here, a number of articles 1507, 1511, 
1515 and 1519 along with their associated titles 1505, 
1509, 1513 and 1517 are displayed in views within a 
window 1501. Generally a major headline 1503 is also 
displayed along with a selection of advertising material 
1521. Because the information provider does not know 
what subjects interest the user, the information provider 
presents a mixture of articles. Here the "New Chemical 
Reaction" article 1 507, and the "Quarks" article 1519 are 
both science and technology related. The other two ar- 
ticles 1515 and 1511 are not. Each article and adver- 
tisement contains information that can be categorized 
in multiple ways. This categorization includes at least 
one topic classifying the information. These topics are 
developed and maintained by the information provider. 
Using gaze tracking, the information provider can deter- 
mine the user's interest in each displayed article 1505, 
1509, 1513 and 1517 and advertisement 1521. Then, 
by using the topics categorizing the presented informa- 
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tion, the information provider can dynamically adjust the 
selection of subsequent information presented to this 
user. In the example above, suppose the user read the 
scientific based articles 1507 and 1519 but did not 
spend any time reading the other articles 1511 or 1515 
or the advertisement 1521. The information provider 
populates the next page of information presented to the 
user with articles and advertisements that have similar 
topics as the previously read information. 

Figure 16 illustrates a possible second page of in- 
formation. Again, the information is provided within 
views contained in a window 1601. Now a plurality of 
articles 1607, 1611, 1615 and 1619 are all scientific or 
technology based, but with different levels of difficulty 
extending from articles of interest to the lay reader to 
those that are directed toward the advanced elemental 
particle physicist. Further, both a Major Scientific Head- 
line 1603 and an advertising 1621 can be selected to be 
of interest to the user. This allows the information pro- 
vider to narrowly target advertising and articles to each 
user. Again the information provider can continue to re- 
fine and narrow the selection of information presented 
to the user on subsequent pages depending on the in- 
terest shown in a plurality of article titles 1605, 1609, 
1613, 1617, the time spent with reading each article 
1607, 1611, 1615 and 1619 and the time spent looking 
at the advertisement 1621 of the current page. 

Figure 17 illustrates the process used to select in- 
formation for presentation to a user. The process starts 
at a terminal 1701 after an initial selection of information 
is displayed to the user. Using the gaze position devel- 
oped as described above, a step 1705 monitors the us- 
er's reading pattern. Further a step 1709, determines 
the amount of interest shown by the user in the dis- 
played information. This interest is determined by meas- 
uring the user's reading speed, determining whether the 
user only skimmed the information or read the informa- 
tion in depth, and by measuring the amount of time spent 
with each article and advertisement. Then in a step 
1711, the process retrieves the topics associated with 
each displayed information and in a step 1713 corre- 
lates these topics with the user's interest. Next in a step 
1715, the process selects additional information based 
on this correlation. This selection of information is dis- 
played in a step 1717 for the user. Finally, the process 
completes through a terminal 1719. In this manner, the 
user is presented with a customized set of information 
that reflects the user's interest. 

Although the present invention has been described 
in terms of the presently preferred embodiments, one 
skilled in the art will understand that various modifica- 
tions and alterations may be made without departing 
from the scope of the invention. Accordingly, the scope 
of the invention is not to be limited to the particular in- 
vention embodiments discussed herein, but should be 
defined only by the appended claims and equivalents 
thereof. 


Claims 

1 . A computer controlled method for displaying a por- 
tion of some of a plurality of articles to a user of a 

s computer; said computer having a display device, 
and a gaze-tracking device; said gaze-tracking de- 
vice determining a gaze position on said display de- 
vice; said computer controlled method comprising 
the steps of: 

10 

(a) displaying said portion of some of said plu- 
rality of articles on said display device; 

(b) detecting an article of interest by determin- 
ing an intersection between said gaze position 

15 and said portion of one of said plurality of arti- 

cles; 

(c) expanding said portion of said article of in- 
terest resulting in an expanded article of inter- 
est; and 

20 (d) displaying said expanded article of interest. 

2. The computer controlled method of claim 1 wherein 
steps (c) and (d) comprise the steps of: 

25 (c) expanding said article of interest by a mag- 

nification factor resulting in said expanded arti- 
cle of interest; 

(d1) displaying said expanded article of inter- 
est; and 

30 (d2) displaying a border enclosing said dis- 

played expanded article of interest. 

3. The computer controlled method of claim 2 wherein 
step (c) further comprises: 

35 

(c1 ) determining a reading speed by monitoring 
said gaze position within said portion of said ar- 
ticle of interest over a period of time; and 
(c2) adjusting said magnification factor to in- 
40 crease said reading speed. 

4. The computer controlled method of claim 1 further 
comprising: 

(e) adjusting said portion of some of said plurality 
45 of articles such that display of said portion does not 
interfere with display of said expanded article of in- 
terest. 

5. The computer controlled method of claim 4 wherein 
50 step (e) further comprises: 

(e1) shifting display of said portion of some of said 
plurality of articles to not interfere with display of 
said expanded article of interest. 

55 6. The computer controlled method of claim 4 wherein 
step (e) further comprises: 

(e1 ) reducing display of said portion of some of said 
plurality of articles to not interfere with display of 
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said expanded article of interest. 

7. The computer controlled method of claim 4 wherein 
step wherein step (c) comprises the steps of: 

(c1 ) determining a reading speed by monitoring 
said gaze position within said portion of said ar- 
ticle of interest over a period of time; 
(c2) adjusting a magnification factorto increase 
said reading speed; and 
(c3) expanding said article of interest by said 
magnification factor resulting in said expanded 
article of interest. 

8. The computer controlled method of claim 1 wherein 
said article of interest is displayed in a view and said 
article of interest comprises textual information; and 
wherein step (c) comprises: 

(c) expanding said textual information within said 
view resulting in said expanded article of interest. 

9. The computer controlled method of claim 8 wherein 
step wherein step (c) comprises the steps of: 

(c1 ) determining a reading speed by monitoring 
said gaze position within said portion of said ar- 
ticle of interest over a period of time; 
(c2) adjusting a magnification factorto increase 
said reading speed; and 
(c3) expanding said textual information within 
said view by said magnification factor resulting 
in said expanded article of interest. 

10. An information presentation apparatus configured 
to display a portion of some of a plurality of articles; 
said information presentation apparatus having a 
central processing unit (CPU), a memory, a display 
device, and a gaze-tracking device; said gaze- 
tracking device capable of determining a gaze po- 
sition on said display device; said information pres- 
entation apparatus comprising: 

a display mechanism configu red to display said 
portion of some of said plurality of articles on 
said display device; 

a detection mechanism configured to detect an 
article of interest by determining an intersection 
between a gaze position and a portion of one 
of said plurality of articles displayed by said dis- 
play mechanism on said display device; and 
an expansion mechanism configured to expand 
said portion of said article of interest resulting 
in an expanded article of interest displayed on 
said display device. 

11. The information presentation apparatus of claim 10 
wherein said expansion mechanism is configured 
to expand said portion of said article of interest by 


a magnification factor resulting in said expanded ar- 
ticle of interest displayed on said display device; 
said expansion mechanism further comprises: 

a border display mechanism configured to 
s display a border enclosing said displayed expanded 
article of interest. 

12. The information presentation apparatus of claim 11 
wherein said expansion mechanism further com- 

10 prises: 

a reading speed determination mechanism 
configured to determine a reading speed by 
monitoring said gaze position within said por- 
15 tion of said article of interest over a period of 

time; and 

a magnification factor adjustment mechanism 
configured to adjust said magnification factorto 
increase said reading speed. 

20 

13. The information presentation apparatus of claim 10 
further comprising: 

an adjustment mechanism configured to ad- 
just said portion of some of said plurality of articles 
25 such that display of said portion does not interfere 
with display of said expanded article of interest. 

14. The information presentation apparatus of claim 13 
wherein said adjustment mechanism further corn- 
so prises an article shifting mechanism configured to 

shift display of said portion of some of said plurality 
of articles to not interfere with display of said ex- 
panded article of interest 

35 15. The information presentation apparatus of claim 1 3 
wherein said adjustment mechanism further com- 
prises an article reduction mechanism configured 
to reduce display of said portion of some of said plu- 
rality of articles to not interfere with display of said 
40 expanded article of interest. 

16. The information presentation apparatus of claim 13 
wherein said expansion mechanism is configured 
to expand said article of interest by a magnification 

45 factor resulting in said expanded article of interest 
displayed on said display device; said expansion 
mechanism further comprises: 

a reading speed determination mechanism 
50 configured to determine a reading speed by 

monitoring said gaze position within said por- 
tion of said article of interest over a period of 
time; and 

a magnification adjustment mechanism config- 
55 ured to adjust said magnification factor to in- 

crease said reading speed. 

17. The information presentation apparatus of claim 10 
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where said article of interest is displayed in a view 
and said article of interest comprises textual infor- 
mation; and said expansion mechanism is config- 
ured to expand said textual information within said 
view resulting in said expanded article of interest 5 
displayed on said display device. 

18. The information presentation apparatus of claim 17 
wherein said expansion mechanism is configured 
to expand said portion of said article of interest by 
a magnification factor resulting in said expanded ar- 
ticle of interest displayed on said display device; 
said expansion mechanism further comprises: 

a reading speed determination mechanism 
configured to determine a reading speed by 
monitoring said gaze position within said por- 
tion of said article of interest over a period of 
time; and 

a magnification adjustment mechanism config- 
ured to adjust said magnification factor to in- 
crease said reading speed. 


a reading speed determination mechanism 
configured to determine a reading speed by 
monitoring said gaze position within said por- 
tion of said article of interest over a period of 
time; and 

a magnification factor adjustment mechanism 
configured to adjust said magnification factor to 
increase said reading speed. 

22. The information presentation system of claim 19 
further comprising: 

an adjustment mechanism configured to ad- 
just said portion of some of said plurality of articles 
such that display of said portion does not interfere 
with display of said expanded article of interest. 

23. The information presentation system of claim 22 
wherein said adjustment mechanism further com- 
prises an article shifting mechanism configured to 
shift display of said portion of some of said plurality 
of articles to not interfere with display of said ex- 
panded article of interest 


15 


19. An information presentation system configured to 
display a portion of some of a plurality of articles; 
said information presentation system having a dis- 
play device and a gaze-tracking device; said gaze- 
tracking device for determining a gaze position on 
said display device; said information presentation 
system comprising: 

a display mechanism configured to display said 
portion of some of said plurality of articles on 
said display device; 

a detection mechanism configured to detect an 
article of interest by determining an intersection 
between a gaze position and a portion of one 
of said plurality of articles displayed by said dis- 
play mechanism on said display device; and 
an expansion mechanism configured to expand 
said portion of said article of interest resulting 
in an expanded article of interest displayed on 
said display device. 

20. The information presentation system of claim 19 
wherein said expansion mechanism is configured 
to expand said portion of said article of interest by 
a magnification factor resulting in said expanded ar- 
ticle of interest displayed on said display device; 
said expansion mechanism further comprises: 

a border display mechanism configured to 
display a border enclosing said displayed expanded 
article of interest. 


24. The information presentation system of claim 22 
25 wherein said adjustment mechanism further com- 
prises an article reduction mechanism configured 
to reduce display of said portion of some of said plu- 
rality of articles to not interfere with display of said 
expanded article of interest. 

30 

25. The information presentation system of claim 22 
wherein said expansion mechanism is configured 
to expand said article of interest by a magnification 
factor resulting in said expanded article of interest 

35 displayed on said display device; said expansion 
mechanism comprises: 

a reading speed determination mechanism 
configured to determine a reading speed by 
40 monitoring said gaze position within said por- 

tion of said article of interest over a period of 
time; and 

a magnification adjustment mechanism config- 
ured to adjust said magnification factor to in- 
45 crease said reading speed. 

26. The information presentation system of claim 19 
where said article of interest is displayed in a view 
and said article of interest comprises textual infor- 

50 mation; and wherein said expansion mechanism is 
configured to expand said textual information within 
said view resulting in said expanded article of inter- 
est displayed on said display device. 


40 


45 


21. The information presentation system of claim 20 55 
wherein said expansion mechanism further com- 
prises: 


27. The information presentation system of claim 26 
wherein said expansion mechanism is configured 
to expand said textual information within said view 
by a magnification factor resulting in said expanded 
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article of interest displayed on said display device; 
said expansion mechanism further comprises: 

a reading speed determination mechanism 
configured to determine a reading speed by 
monitoring said gaze position within said por- 
tion of said article of interest over a period of 
time; and 

a magnification adjustment mechanism config- 
ured to adjust said magnification factor to in- 
crease said reading speed. 

28. A computer program product comprising: 

a computer usable storage medium having 
computer readable code embodied therein for 
causing a computer to display a portion of some 
of a plurality of articles on a display device; said 
computer readable code comprising: 
computer readable code devices configured to 
cause said computer to effect a gaze position 
determination mechanism configured to deter- 
mine a gaze position from gaze coordinates re- 
turned from a gaze-tracking device; 
computer readable code devices configured to 
cause said computer to effect a display mech- 
anism configured to display said portion of 
some of said plurality of articles on said display 
device: 

computer readable code devices configured to 
cause said computer to effect a detection 
mechanism configured to detect an article of in- 
terest by determining an intersection between 
said gaze position and said portion of one of 
said plurality of articles displayed by said dis- 
play mechanism on said display device; and 
computer readable code devices configured to 
cause said computer to effect an expansion 
mechanism configured to expand said portion 
of said article of interest resulting in an expand- 
ed article of interest displayed on said display 
device. 

29. The computer program product of claim 28 wherein 
said expansion mechanism is configured to expand 
said portion of said article of interest by a magnifi- 
cation factor resulting in said expanded article of in- 
terest displayed on said display device; said expan- 
sion mechanism further comprises: 

computer readable code devices configured 
to cause said computer to effect a border display 
mechanism configured to display a border enclos- 
ing said displayed expanded article of interest. 

30. The computer program product of claim 29 wherein 
said expansion mechanism further comprises: 

computer readable code devices configured to 


cause said computer to effect a reading speed 
determination mechanism configured to deter- 
mine a reading speed by monitoring said gaze 
position within said portion of said article of in- 
s terest over a period of time: and 

computer readable code devices configured to 
cause said computer to effect a magnification 
factor adjustment mechanism configured to ad- 
just said magnification factor to increase said 
10 reading speed. 

31. The computer program product of claim 28 further 
comprising: 

computer readable code devices configured 
15 to cause said computer to effect an adjustment 
mechanism configured to adjust said portion of 
some of said plurality of articles such that display of 
said portion does not interfere with display of said 
expanded article of interest. 

20 

32. The computer program product of claim 31 wherein 
said adjustment mechanism further comprises 
computer readable code devices configured to 
cause said computer to effect an article shifting 

25 mechanism configured to shift display of said por- 
tion of some of said plurality of articles to not inter- 
fere with display of said expanded article of interest 

33. The computer program product of claim 31 wherein 
30 said adjustment mechanism further comprises 

computer readable code devices configured to 
cause said computer to effect an article reduction 
mechanism configured to reduce display of said 
portion of some of said plurality of articles to not in- 
35 terfere with display of said expanded article of in- 
terest. 

34. The computer program product of claim 31 wherein 
said expansion mechanism is configured to expand 

40 said article of interest by said magnification factor 
resulting in an expanded article of interest displayed 
on said display device; said expansion mechanism 
comprises: 

45 computer readable code devices configured to 

cause said computer to effect a reading speed 
determination mechanism configured to deter- 
mine a reading speed by monitoring said gaze 
position within said portion of said article of in- 
50 terest over a period of time: and 

computer readable code devices configured to 
cause said computer to effect a magnification 
adjustment mechanism configured to adjust 
said magnification factor to increase said read- 
55 ing speed. 

35. The computer program product of claim 28 where 
said article of interest is displayed in a view and said 
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article of interest comprises textual information; and 
wherein said expansion mechanism is configured 
to expand said textual information within said view 
resulting in said expanded article of interest dis- 
played on said display device. s 

36. The computer program product of claim 35 wherein 
said expansion mechanism is configured to expand 
said textual information within said view by a mag- 
nification factor resulting in said expanded article of 10 
interest displayed on said display device; said ex- 
pansion mechanism further comprises: 

computer readable code devices configured to 
cause said computer to effect a reading speed 15 
determination mechanism configured to deter- 
mine a reading speed by monitoring said gaze 
position within said portion of said article of in- 
terest over a period of time; and 
computer readable code devices configured to 20 
cause said computer to effect a magnification 
adjustment mechanism configured to adjust 
said magnification factor to increase said read- 
ing speed. 
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